Automated wayside asset monitoring with optical imaging and visualization

ABSTRACT

The Automated Wayside Asset Monitoring system utilizes a camera-based optical imaging device and an image database to provide intelligence, surveillance and reconnaissance of environmental geographical information pertaining to railway transportation. Various components of the Automated Wayside Asset Monitoring system can provide features and functions that can facilitate the operation and improve the safety of transportation via a railway vehicle.

PRIORITY CLAIMS

This application is a continuation of U.S. patent application Ser. No.16/215,342, filed on Dec. 10, 2018, which is a continuation of U.S.patent application Ser. No. 15/631,381, filed on Jun. 23, 2017, andissued as U.S. Pat. No. 10,179,597 on Jan. 15, 2019, which claims thebenefit of U.S. Provisional Patent Application No. 62/355,053, filed onJun. 27, 2016, the contents of which are incorporated herein byreference.

BACKGROUND OF THE INVENTION

The present disclosure relates to transportation infrastructuregenerally, and more particularly, to methods and systems forvehicle-centric railway wayside asset monitoring and systemoptimization.

The worldwide demand for passenger and freight travel is expected todouble by 2050, compared with 2010 levels requiring an additional200,000 miles of rail track. Meeting this demand will require increasedtrain speeds, loads and frequency, adding stress on the aging railwayinfrastructure. Railroads and regulatory bodies have adopted programs toaddress growing safety needs. One of these initiatives is the PositiveTrain Control (PTC) regulation resulting from the United States RailwaySafety Act of 2008 (RSIA). These safety initiatives, coupled with a needfor improved operations, require a recurrent mapping and monitoring ofrailroad assets.

In the United States, the Rail Safety Improvement Act of 2008 requirespassenger railroad and Class I railroad to install positive traincontrol (PTC) on main lines used to transport passengers ortoxic-by-inhalation hazardous materials. One portion of this requiremententails mapping over 60,000 miles of right-of-way and 476,000 assets.

The asset database, including critical features such as the presence ofsignals and switches, must be validated asset-by-asset and mile-by-mileat regular intervals. The railroads must ensure that what is displayedto the train crew via the track database and onboard system reflectswhat is shown by railroad signal and what is actually present on theground. Furthermore, construction and validation of the rail assetdatabase is a continuous process since almost all changes to therailroad infrastructure require its modification and re-validation.Validation of the database is a time-consuming and labor-intensiveprocess. What is needed is a system to create, update, validate andaccess information about the location and status of wayside assets.

With respect to geographic information systems (GIS), the accuracy ofthe information required for PTC is significantly more precise than whatis required to run a safe and efficient railroad in a non-PTCenvironment. Today, there are approximately 500,000 critical assets thatmust be geo-located to a horizontal precision of less than 2.2 meters(˜7 feet) and a vertical precision of 0.8 m (˜2 feet) to provide theaccuracy necessary to safely warn or stop a locomotive. Additionally,yards, industry, and other connecting track must also be mapped toaccount for entry onto and exit from PTC track totaling more than 63,000miles of right-of-way.

There are substantial challenges to developing and implementing asustainable process to document and update the location every time oneof the over 460,000 critical PTC assets is moved by more than 1 foot.Updating the PTC track database is a continuous process since almost allchanges that occur in the railroad infrastructure require reconstructionand revalidation.

In terms of the PTC initiative, one of the most challenging aspects isthe requirement for interoperability and real-time communication betweenthe various system components. This is in part due to the fact that thecomponent systems are developed by various suppliers, and theimplementations by each differ across the various railroad companies.However, there have been unforeseen challenges in retrofitting an agingrailroad infrastructure that was not designed with today's technologiesin mind. All these factors have led to significant program delays andhave raised questions about the practicality of the current approach. Inlight of such challenges, the ability to implement one of the componentsystems, the track database in a manner that is not strictly dependenton the operation of other components subsystems offers a way forward toimproving safety and advancing progress toward meeting regulatoryrequirements.

SUMMARY OF THE INVENTION

This system for wayside asset monitoring comprises a portable camerasystem that mounts to a vehicle (track geometry car, Hi-Rail truck,unmanned aerial vehicle, etc.) that travels linearly along the trackcollecting optical camera images of wayside assets and tagging eachimage with metadata such as geo-location, etc. The data is stored by thecamera system and subsequently processed to identify critical assets andcreate and/or update and validate the track database. The database maybe accessed by train operators during trips using an onboard computerthat runs a software dashboard to provide them information that improvessituational awareness. The system does not require any modification totrains. Nor does it require installation of GPS antennas or any otherspecial equipment or markings at wayside locations.

The present invention relates to Intelligence, Surveillance andReconnaissance (“ISR”) optical imaging and visualization technology. Thesystem comprises of methods that utilize ISR imaging as a means of assetmonitoring through cameras mounted to the roof of a hi-rail vehicle inorder to detect and validate specific assets. The system also comprisesof a database that stores the asset monitoring data captured by thecamera. This database is used within the situational awareness dashboardwhere high resolution video wayside imagery is captured for the assetmonitoring system to augment the visual acuity of the long haullocomotive engineer and can therefor improve the safe operation of thetrain in degraded visual environments.

present invention automates the task of monitoring railway waysidesignals, signage, bridges, tracks, guardrails, switches, foliage, etc.The system uses a high-resolution camera array mounted on a hy-rail tooptically surveil the wayside at speed without human intervention. Thecamera array collects high-resolution 360° video of the wayside toidentify and track assets to validate location, visual appearance, andother important information required for regulatory compliance, safety,and operational efficiency. Wayside clearance and restricted areas aremonitored for degradation and intrusive structures. The camera systemupdates a video archive with GPS-synchronized metadata for processingagainst a track database. Assets with detected anomalies are tagged forfollow-up operator inspection using a desktop software application thatautomatically validates the track database. The system employs deeplearning algorithms with convolutional neural networks (CNN) for objectidentification and change detection.

As a train proceeds down the track its GPS coordinates are used toaccess a video archive and present a synchronized view from a previousrun. The operator can press a button and see a side-by-side video ofwhat the camera is seeing now versus what it saw previously at the sameexact spot—particularly helpful during periods of degraded visibility. A“Look Ahead” capability can stream video of what's around the corner, 1,2 miles or minutes ahead to increase situational awareness and be betterprepared for travel through high-risk areas. Advanced algorithms canaugment human perception and response by processing camera imagery inreal time. Real-time analytics can mitigate crew visual fatigue, enhancesafety awareness, and warn of dangers not discernible by the human eye.The system can be configured to keep operators advised of specialconditions and requirements defined by GPS coordinates, operatingconditions, or events.

When an event occurs, the Dashboard can provide procedural and policyguidance. For example, the system might be configured to monitoradherence to signals and signs, or advise when quiet zones hours are ineffect, or alert when shrubbery exceeds the height limit. A computerscripting language is available for system configuration and management.The scripting language is used to describe the desired system behaviorat a particular geolocation. It is used to define specific protocols forthe crew to follow in response to certain wayside signals or anomalousconditions.

The camera system includes an array of cameras configured to collect ahigh-resolution 360° video of the wayside. When performing audit and/orsurvey tasks object detection can be uni-directional meaning that thesystem detects critical assets facing against the direction of travel oralternatively detection can be bi-directional meaning that the systemdetects critical assets facing both against the direction of travel andwith that direction. Examples of types of objects tracked include SpeedSigns category, Train Control Signals category, and Milepost objects.

The camera system incorporates proprietary image processing technologythat improves visibility in imagery captured in degraded visionenvironment such a fog, blinding light, etc. Optical image processingand object recognition techniques are applied to the image framescaptured by the camera system to find and classify objects and tomeasure with precision and to build and validate a database. The systememploys various technologies to do this including machine vision andobject recognition and tracking. Object recognition can be implementedusing Convolutional Neural Network (CNN) technology whereby the systemcan be trained to identify specific objects by analyzing a large sampleof similar objects. Vision-based neural networks attempt to emulate theway a human brain works to recognize specific objects by passing thevisual input through a series of many layers for processing. In humans,images flow from the retina into the primary visual cortex and theninfrared temporal (IT) cortex. In neural networks the visual data ispassed through a series of mathematical models, each specialized inidentifying a visual element such as a dot or a line. The mathematicalmodel for an individual element can be very simple, but when thousandsand millions of these are combined it is possible to get very complextransformations from the raw signals that yield representations that arevery good for object recognition. Neural networks are “trained” toidentify certain objects by using machine-learning algorithms to analyzedatasets that contain very large numbers of images, each one of which isannotated by humans with different levels of identification. Over time,the system learns as it is exposed to more and more images and becomesincreasingly accurate and reliable in recognizing specific types ofobjects.

The system includes object tracking and identification technology thatrecognizes specific types of objects at specific locations. Assets withdetected anomalies are tagged for follow-up operator inspection using adesktop software application that automatically validates the trackdatabase.

A Situational Awareness Dashboard software application uses informationfrom the database to provide train operators tools and information toimprove situational awareness and decision making. In conditions oflimited visibility, train operators can view side-by-side image streamsto compare real-time trip imagery with recorded imagery of the previoustrip to gain a sense of the lay of the land. When asset anomalies aredetected by the system, procedural and policy information can bepresented to the train operator to assist them. The SituationalAwareness Dashboard could monitor adherence to signals and signs oradvise when quiet zones hours are in effect so that the horn isn'tsounded.

These and other features and aspects of the disclosed technology willbecome apparent from the following detailed description, taken inconjunction with the accompanying drawings, which illustrate, by way ofexample, the features in accordance with embodiments of the disclosedtechnology. The summary is not intended to limit the scope of anyinventions described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of the rail asset monitoring systemsensor mounted to the roof of a hi-rail vehicle.

FIG. 2 is a schematic illustration of the top view of the mounted railasset monitoring system sensor.

FIG. 3 is a schematic illustration of the hardware used in rail assetmonitoring system sensor.

FIG. 4 is a block diagram of the monitoring system data collection andaggregation process.

FIG. 5 is an illustrated representation of the situational awarenessdashboard interface.

FIG. 6 is an illustrated representation of how wayside rail assetmonitoring data is presented on the back-end work station where a humanoperator can review and validate the information before the assetdatabase is updated.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 is a schematic illustration of the rail asset monitoring systemsensor mounted to the roof of a hi-rail vehicle. In accordance with thepreferred embodiment of the present invention, wayside assets aretracked using an off-track vehicle such as a hi-rail truck equipped withan apparatus that includes one or more cameras to capture image data andmetadata of the assets and generate an image stream. The apparatus isdesigned to be portable such that it can be quickly mounted to a vehicleand perform surveillance, image collection, and object identificationand verification functions at high speeds appropriate to the vehicle andpathway.

FIG. 2 is a schematic illustration of the top view of the mounted railasset monitoring system sensor. In accordance with the preferredembodiment of the present invention, the camera apparatus is enclosed asealed chassis the ingress protection rated IP66. The enclosedelectronics conform to railroad industry standards and specification.The chassis is equipped with a removable quick-connect/disconnect systemfor mounting to the roof of Ford F-150 truck or similar vehicle. Thecamera lens windows are protected with changeable shields to avoidscratches and dings, etc. A cable from the apparatus enters the vehiclecab through a window seal to provide the DC power, HDMI, and USBconnectivity to a laptop inside the cab so a passenger can view thecamera feeds and interact with the system.

In one embodiment, five miniature cameras are connected to the chassisvia Power over Ethernet and the apparatus can be magnetically mounted tothe top or sides of the vehicle to provide the best view. Five camerasare used rather than four to avoid image warping when the camera imagestreams are combined in a mosaic fashion to create a 360-degree viewaround the truck.

In one embodiment the apparatus controller houses a GigE network switchthat connects to the cameras, one image processing system board for eachof the camera, an HDMI quad mixer chip to combine the individualprocessed camera image streams into a single image stream and an HDMIconnector to output the image stream. The chassis includes a powersupply, an IMX processor, a real-time clock, a GPS receiver, a 6-axisMEMS accelerometer/gyroscope device, an acoustic sensor, an ambientlight sensor, a H.264 encoder and a removable solid state drive (“SSD”).

The individual image processing boards for each camera include an IMXprocessor for decoding the camera image stream and for insertingmetadata from other sensors and an FPGA that applies specializedalgorithms for image enhancement. After the camera image stream isprocessed it is pushed to the controller's IMX processor via the GigEswitch and also a mirrored stream is output via HDMI so the operator canview it.

Image streams captured by the camera apparatus are stored on theremovable store device. The storage device can be removed from thecamera apparatus and connected to a computer system for post-collectionprocessing of the image stream as described above.

FIG. 3 is a schematic illustration of the hardware used in rail assetmonitoring system sensor. In accordance with the preferred embodiment ofthe present invention, the basic system contains an electronics modulethat manages and synchronizes multiple cameras and sensors to surveilthe way forward and track wayside. Baseline functionality providesreal-time image processing to enable cameras to see through visualobfuscation while building a database of GPS-synchronized images. Imagestreams from multiple HD cameras can be combined into a single streamwith multiple windows, such as quad-view. The cameras can include anoptional Time of Flight (“ToF”) Sensor to measure distance. The camerasmay be removable and can be positioned on top of or inside of thevehicle to provide up to a 360-degree field of view. Image streams fromthe camera can be a series of still images frames with locationmetadata, rather than a conventional video stream.

The apparatus includes a GPS capability and geo-location metadata thatcan be included in the image stream. The GPS can be continuallyrecalibrated using geo-locations of known fixed structures identified inthe database and Real Time Kinematic (“RTK”) satellite navigation.

The apparatus includes image processing technology that improvesvisibility in degraded vision environment such a fog or blinding light.

The apparatus includes object tracking and identification technologythat recognizes specific types of objects at specific locations. Theapparatus includes a storage device for retaining data received from thevarious sensors and stores image streams for later processing. Thestorage devices can be a removable drive or removable media so that datacan be transported to another system for further processing.

The apparatus can include an audio sensor and audio metadata can beincluded in the image stream. The audio sensor can be used to detecthorns, crossings bells, quiet zones.

The apparatus can include a MEMS inertial accelerometer and gyroscopicsensor and positional metadata can be included in the image stream. MEMSsensor can be used to analyze grade, tilt, acceleration, speed, andother important criteria.

The apparatus can include a wireless communication module. Wirelesscommunication can be used to: transmit image streams and metadata; tosynchronize a local copy of the database with a remote version; toprovide system command and control information.

The apparatus can include an ambient light sensor. Ambient light sensorscan be used to validate areas requiring lighting such as tunnels andbridges.

FIG. 4 is a block diagram showing the flow of image data through themonitoring system data collection and aggregation process. In accordancewith the preferred embodiment of the present invention, the apparatuscan automatically identify, record, and verify the geo-location andstatus of wayside objects as it moves along railroad tracks, streets andhighways, or other vehicle pathways. The purpose of the system is to:create an initial database of objects of interest identified along awayside, and; thereafter, using a previously created database, confirmthe position and status of previously identified objects, and; add newobjects to an existing database. Image streams can be viewed duringcollection by an operator in the vehicle and potentially manuallyprocessed. However, this is optional and the preferred method is topost-process the collected data electronically or with operatorassistance to changes and update the database. The system can also beused for optical surveillance and monitoring for situational awareness,collision warnings, infrastructure inspection, and safety or riskmitigation.

Optical image processing and object recognition using image frames areused to find and classify the object and to measure with precision andto use the optics to validate a database or to build a database. Varioustechnologies including machine vision and object recognition andtracking are implemented in the system's software and/or hardware.Object recognition can be implemented using Convolutional Neural Network(CNN) technology whereby the system can be trained to identify specificobjects by analyzing a large sample of similar objects. Or otheralgorithms and approaches such as feature-based similarity search in 3Dobject databases may be used.

A computer scripting language can be used for system management todefine protocols to execute in response to wayside signals or ananomalous condition in the wayside. For each geo-location, the railroadcompany can provide a program using the scripting language to describethe desired system behavior at a particular geo-location. Look for this;measure that. Is it this color? Is at this distance? Is it making thisnoise?

FIG. 5 is an illustrated representation of the Situational AwarenessDashboard User Interface. In accordance with the preferred embodiment ofthe present invention, a Situational Awareness Dashboard softwareapplication uses information from the database to provide the train crewtools and information to improve situational awareness and decisionmaking. In conditions of limited visibility, crew members can viewside-by-side image streams to compare real-time trip imagery withrecorded imagery of the previous trip to gain a sense of the lay of theland. When asset anomalies are detected by the system, procedural andpolicy information can be presented to the crew to assist them. TheSituational Awareness Dashboard could monitor adherence to signals andsigns or advise when quiet zones hours are in effect so that the hornisn't sounded, etc.

FIG. 6 is an illustrated representation of how wayside rail assetmonitoring data is presented on an Operator Workstation, which is partof the back-end processing software that may be associated with thepresent invention. In accordance with the preferred embodiment of thepresent invention, a database is used to track wayside assets and thedatabase can be validated and updated using image data and metadata fromthe apparatus. If an asset database does not yet exist, the image datacan be used to create a new database and populate it. The database canbe generated automatically using the computer system. A human operatorcan interact with the computer system to view image data, check resultsor make changes to the database.

If an asset database already exists, the image data can be used tovalidate and update the database. The database can be updated andvalidated automatically using a computer system that compares thepresence or absence of identified assets at designated locations. Ahuman operator can interact with the computer system to view image data,check results or make changes to the database. The system can generatereports such as a report showing a list of items in the database thathave changed. The database contains descriptions of objects of interestthat are based on one or more previously captured images, representativesimilarity images, descriptive metrics, and mathematical models.

The database of objects contains the object's name, type, geo-locationusing GPS latitude and longitude, horizontal and vertical distance fromthe vehicle, pathway or known reference point, a history of imagescollected, a list of physical and visual characteristics and referenceimages used to identify the object, pointers to executableuser-programmable scripts that describe how to validate the object'sstatus, and how to report the verification results. The databasesupports a new scripting language that allows users to describe objectsof interest in detail and to specify actions to be taken when certainobjects are identified by the systems or marked as absent.

Although the disclosed technology is described above in terms of variousexemplary embodiments and implementations, it should be understood thatthe various features, aspects and functionality described in one or moreof the individual embodiments are not limited in their applicability tothe particular embodiment with which they are described, but instead maybe applied, alone or in various combinations, to one or more of theother embodiments of the disclosed technology, whether or not suchembodiments are described and whether or not such features are presentedas being a part of a described embodiment. Thus, the breadth and scopeof the technology disclosed herein should not be limited by any of theabove-described exemplary embodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent. The use of theterm “module” does not imply that the components or functionalitydescribed or claimed as part of the module are all configured in acommon package. Indeed, any or all of the various components of amodule, whether control logic or other components, may be combined in asingle package or separately maintained and can further be distributedin multiple groupings or packages or across multiple locations.

Additionally, the various embodiments set forth herein are described interms of exemplary block diagrams, flow charts and other illustrations.As will become apparent to one of ordinary skill in the art afterreading this document, the illustrated embodiments and their variousalternatives may be implemented without confinement to the illustratedexamples. For example, block diagrams and their accompanying descriptionshould not be construed as mandating a particular architecture orconfiguration.

Embodiments presented are particular ways to realize the invention andare not inclusive of all ways possible. Therefore, there may existembodiments that do not deviate from the spirit and scope of thisdisclosure. It will be appreciated that a great plurality of alternativeversions are possible.

1. A railroad management system, comprising: a display device configuredto display information to an operator; and a controller in electroniccommunication with display device and a central control device andincluding: a train control module configured to receive input data fromone or more train control systems and generate command signals forcontrolling operations of a train based on the input data from the oneor more train control systems in response to said central controldevice; a business systems module configured to receive input data fromone or more railroad data management systems and for providingperformance data to said central control device; and an integrationmodule configured to: identify discrepancies among the input data fromthe one or more train control systems and the railroad data managementsystems and for reporting said discrepancies to said central controldevice; and generate corrected input data based on the input data fromthe one or more train control systems and the railroad data managementsystems; wherein the controller is configured to display one or moregraphical user interfaces on the display device based at least in parton the corrected input data and causes a change of position of railroadassets in response to a cumulative collection of said discrepancies. 2.The railroad management system of claim 1 wherein real time images arecompared with historic images to ascertain changed conditions.
 3. Therailroad management system of claim 2 herein real time images arecompared with a plurality of historic images to ascertain changedconditions over a period of time.
 4. The railroad management system ofclaim 2 wherein said changed conditions prompt a railroad maintenancerequest for attention.
 5. The railroad management system of claim 3herein said changed conditions prompt a railroad maintenance request forattention.
 6. A method for managing a railroad system, comprising: adisplay device configured to display information to an operator; and acontroller in electronic communication with display device and a centralcontrol device and including: a train control module configured toreceive input data from one or more train control systems and generatecommand signals for controlling operations of a train based on the inputdata from the one or more train control systems in response to saidcentral control device; a business systems module configured to receiveinput data from one or more railroad data management systems and forproviding performance data to said central control device; and anintegration module configured to: identify discrepancies among the inputdata from the one or more train control systems and the railroad datamanagement systems and for reporting said discrepancies to said centralcontrol device; and generate corrected input data based on the inputdata from the one or more train control systems and the railroad datamanagement systems; wherein the controller is configured to display oneor more graphical user interfaces on the display device based at leastin part on the corrected input data and causes a change of position ofrailroad assets in response to a cumulative collection of saiddiscrepancies.
 7. The method for managing a railroad system of claim 6wherein real time images are compared with historic images to ascertainchanged conditions.
 8. The method for managing a railroad system ofclaim 7 wherein real time images are compared with a plurality ofhistoric images to ascertain changed conditions over a period of time.9. The method for managing a railroad system of claim 7 wherein saidchanged conditions prompt a railroad maintenance request for attention.10. The method for managing a railroad system of claim 8 wherein saidchanged conditions prompt a railroad maintenance request for attention.11. A railroad maintenance system, comprising: a display deviceconfigured to display information to an operator; and a controller inelectronic communication with display device and a central controldevice and including: a train control module configured to receive inputdata from one or more train control systems and generate command signalsfor controlling operations of a train based on the input data from theone or more train control systems in response to said central controldevice; a business systems module configured to receive input data fromone or more railroad data management systems and for providingperformance data to said central control device; and an integrationmodule configured to: identify discrepancies among the input data fromthe one or more train control systems and the railroad data managementsystems and for reporting said discrepancies to said central controldevice; and generate corrected input data based on the input data fromthe one or more train control systems and the railroad data managementsystems; wherein the controller is configured to display one or moregraphical user interfaces on the display device based at least in parton the corrected input data and causes a change of position of railroadassets in response to a cumulative collection of said discrepancies andwherein said discrepancies activate maintenance orders for maintainingrailroad operations.
 12. The railroad management system of claim 11wherein said maintenance orders are acted upon based on a schedule ofcosts and budgets pertaining to said railroad.
 13. The railroadmanagement system of claim 12 wherein maintenance procedures may beprioritized based on relevant safety concerns.